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Evidence-based improvement in higher education 



Reform has been on the higher education agenda for at least a quarter-century. The 
National Institute of Education’s 1984 report, Involvement in Learning, proposed research on 
improving educational quality, such as: “What are the most effective organizational strategies, 
policies, and processes available to administrators as they seek to maximize student learning and 
development and simultaneously to utilize their resources more efficiently?” The report also 
emphasized the goal to improve educational quality and challenged institutions to adopt 
organizational strategies, policies, and processes that would maximize the use of evidence to 
inform institutional improvement. 

Calls to improve undergraduate education have grown more insistent in recent years. 
Stakeholders are demanding greater accountability and transparency, and expressing doubts 
about the quality of undergraduate education (e.g.. Commission on the Future of Higher 
Education, 2006). In response to the push for accountability and evidence of quality, many 
colleges and universities have launched significant reform initiatives (Kezar & Eckel 2002). In 
addition, most institutions of higher education have increased their assessment activities and are 
using this information to inform campus change efforts and as a means for monitoring progress 
(Banta, 2009; Kuh & Ikenberry, 2009; Maki, 2004). However, while there has been increased 
activity in assessment and reform efforts at colleges and universities, there is less evidence that 
the assessment loop has been closed — examining whether the changes undertaken have produced 
improvements (Banta, 2009; Suskie, 2004). In fact, few studies examine before-and-after 
assessment results to substantiate institutional change (Keup et al. 2001; Jeffery 2008). Also 
missing are qualitative case studies that analyze the process of change (Astin et al. 2001; Keup et 
al. 2001). The research project of which this paper is a part aims to fill these gaps by using time 
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series data from a widely used assessment to (1) identify institutions that demonstrate positive 
trends on a range of educational effectiveness measures, (2) describe observed patterns of 
improvement, and (3) examine informants’ accounts of what produced the improved results. This 
study informs broad questions about organizational learning and educational improvement by 
identifying the circumstances that enable intentional change to take root and thrive. 

This research project takes advantage of a well-established national assessment initiative, 
the National Survey of Student Engagement (NSSE), that provides participating colleges and 
universities with valuable information about educational practices associated with desired 
learning outcomes. More than 1,400 baccalaureate degree-granting colleges and universities 
participated in NSSE from 2000 through 2009, representing the full range of institutional types. 
As of 2009, more than 530 U.S. institutions had participated at least four times, with many 
having participated five, six, or more times. NSSE surveys random samples of first- year and 
senior students to determine the time and effort they put into their studies and their involvement 
in a range of activities associated with valued educational outcomes (Chickering & Gamson, 
1987; Kuh, 2001). NSSE also assesses the organization of learning opportunities and services to 
promote learning and success. Because NSSE focuses on actionable information, results can 
inform efforts to improve undergraduate education. Standardized sampling and administration 
protocols assure that results are comparable between institutions and over time. NSSE thus 
provides a rich source for studying both the prevalence of effective practices and institution-level 
trends across a wide range of institutions. 

The availability of institution-specific results over several years affords a unique 
opportunity to identify cases (institutions) that show positive trends in effective practices and to 
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investigate what contributed to these results. In short, NSSE provides a window into 
improvement and organizational learning in U.S. colleges and universities. 

Our specific research questions are: 

1. Do NSSE results provide evidence of trends (positive or negative) in effective 
educational practices at the institutional level? 

2. If trends are found, which measures or combinations most commonly show trends? 

3. How do institutional informants account for positive trends? 

Conceptual framework 

The study is informed by treatments of organizational learning and intentional change in 
organizations. Organizational learning provides a framework for understanding how 
organizations acquire and interpret information, interpret their experience, and make choices, 
while the literature on change provides a sharper focus on goal-directed change. 

Eundamental propositions about organizational learning hold that learning results from 
interpreting experience and is encoded into organizational routines and standard operating 
procedures (Daft & Weick 1984; Eiol & Eyles 1985; Eevitt & March 1988). Among the 
conceptual differences expressed by theorists of organizational learning is whether it is 
necessarily positive and produces improved performance. Contrast Huber’s agnostic view — 
“[E]eaming does not always increase the learner's effectiveness or even potential effectiveness. . . 
Entities can incorrectly learn, and they can correctly learn that which is incorrect” (1991, p. 

89) — with the improvement-focused orientation of Eiol and Eyles: “Organizational learning 
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means the process of improving actions through better knowledge and understanding” (1985, p. 
803). From an evaluation perspective, Torres and Preskill (2002) see organizational learning as 
involving intentional, goal-directed change: “Organizational learning is a continuous process of 
growth and improvement that (a) uses information or feedback about both processes and 
outcomes (i.e. evaluation findings) to make changes; (b) is integrated with work activities, and 
within the organization’s infrastructure (e.g., its culture, systems and structures, leadership, and 
communication mechanisms); and (c) invokes the alignment of values, attitudes, and perceptions 
among organizational members” (p.388). We embrace the latter perspectives that learning is 
improvement- focused. 

Some theorists posit an important role for organizational interpretations of the past and 
the information that an organization collects (Cohen & Sproull 1991; Levitt & March 1988; Daft 
& Weick 1984). This study calls special attention to the interpretive function, given our interest 
in the use of assessment data to inform improvement efforts. 

Levitt and March observe that organizations evaluate performance relative to targets, and 
that their reaction “depends on the relation between the outcomes they observe and the 
aspirations they have for those outcomes” (1988, p. 320). Similarly, Argyris and Schon (1978) 
view organizational learning as arising from “a surprising mismatch between expected and actual 
results” that stimulates a desire for change. This is consistent with Schein’s perspective that “all 
forms of learning and change start with some form of dissatisfaction or frustration generated by 
data that disconfirm our expectations or hopes” (1996). We assume that assessment data is a 
potential source of such disconfirmation, motivating action to render actual experience more 
consonant with aspirations, hopes, and expectations. Our focus on intentional change assumes 
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that we will find cases where assessment results were disappointing, leading to interventions to 
improve future performance. However, we also recognize that we may encounter cases where 
positive results are unplanned or emergent, resulting perhaps from uncoordinated local efforts 
(Leslie, 1996). 

Studies of organizational learning in education are rare, particularly in higher education. 
Dill (1999) studied the adaptation of a multinational sample of universities to new quality 
assurance regimes using a framework that based on the organizational learning literature. He 
concluded that five features characterize an “academic learning organization:” a culture of 
evidence, meaning a commitment to systematic use of evidence in problem identification and 
solution; improved coordination of teaching units; learning from others; the establishment of 
structures to promote and support the improvement of teaching and learning; and knowledge 
transfer between academic units within the university. 

In a qualitative study of teams at 14 campuses participating in the Diversity Scorecard 
project, Bauman (2005) concluded that three conditions contributed to organizational learning 
(defined as a new recognition of institutional problems related to educational equity): the 
presence of new ideas — that is, new ways of looking at institutional data; raising doubts about 
knowledge and practices, thereby questioning existing routines, norms, and shared 
understandings; and development and transfer of new knowledge among institutional actors. 

The literature on change in institutions of higher education demonstrates that reform is as 
dynamic and complex as it is in any other organization (Kezar, 2001). Hearn (1996) asserted 
several propositions about transformation in U.S. higher education, including insights that 
organizational politics and institutional culture are important considerations for change, and that 
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transformation is resource-dependent. Many studies of change in higher education have 
demonstrated that the existing structures of colleges and universities, including loosely coupled 
systems, shared governance, and employee commitment, affect the change process. In addition, 
change models for institutions of higher education include an array of variables such as the 
environment, political climate, the commitment of leaders, and the involvement of the whole 
system (Kezar, 2001). Implementing large-scale, transformational change in colleges and 
universities is difficult, due in part to their complexity and also to their governance practices 
(Cuban, 1999; Birnbaum, 2000). Overall, research on institutional change suggests that 
“institutional transformation” is rare and that, if change is possible, it is most likely to be 
incremental (Kezar, 2001). Models of institutional change in colleges and universities are 
complex, take many forms, and are highly dependent on the type of change being initiated. 

Data and methods 

From the outset, one of NSSE’s principal goals has been to provide participating colleges 
and universities with diagnostic, actionable information that can be used to improve 
undergraduate education. This goal, combined with steady growth in the number of participating 
institutions and large numbers that administer the survey on a regular basis, means the NSSE 
data archive is well-suited to examining questions of institutional change. To conduct a complete 
examination of trends and to understand how institutions account for change, we designed two 
distinct research phases. Phase one involved the examination of trends across a variety of 
measures of educational effectiveness among institutions with multiple NSSE administrations, 
affording a longitudinal view of change and stability in NSSE results. This entailed quantitative 
analysis of time series data from institutions that participated in NSSE multiple times over a 
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nine-year period. Phase two involved a qualitative approach to study a select subset of 
institutions that displayed positive trends to learn more about their change initiatives (if any), 
actors involved, and interpretations of what contributed to the improvements. 

Phase One: Detecting Trends 

For the first phase of the study, we identified 534 institutions that had administered NSSE 
at least four times between 2001 and 2009. Two-thirds of this sample had at least five 
administrations, and one-quarter had seven or more (table 1). The sample reflects the diversity of 
U.S. higher education with respect to size, control, Carnegie classification, and region. 
Specifically, the sample comprised 18% Doctorate-granting, 46% Master’s, 33% Baccalaureate, 
and 3% other types of colleges and universities; 43% of the institutions were public and 7% were 
minority- serving institutions. With regard to undergraduate enrollment, 27% were very small 
(fewer than 2,000 students), 32% were small (2,000 - 4,999), 19% were medium (5,000 - 
9,999), and 23% were large (at least 10,000). 

Because NSSE surveys first-year and senior student populations, we analyzed trends 
separately for each group. The first-year student dataset included more than 600,000 individuals, 
of whom 56% were female and 77% were age 19 or younger. The senior dataset included more 
than 640,000 students (58% female and 63% under age 24). The average institutional response 
rate was 37.9%, and the average number of respondents per institution was 282 first-year and 
304 senior students. 

Eor phase one, we examined the following measures: four NSSE Benchmarks of 
Effective Educational Practice (Academic Challenge, Active & Collaborative Eeaming, Student- 
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Faculty Interaction, Supportive Campus Environment); six subscales (active learning, 
collaborative learning, course-related faculty interaction, out-of-class faculty interaction, support 
for success and enrichment, support for academic success); two course emphasis scales (higher 
order thinking and integrative learning); a scale tapping experiences with diversity; and the 
proportion of students reporting high-impact practices (see Kuh, 2008). We examined measures 
separately for first- year students and seniors. Additional detail about these measures is presented 
in the appendix. 

First, we identified and excluded instances of unreliable data for a given institution and 
year (that is, administrations with low response rates or large sampling errors). For example, we 
excluded administrations with sampling errors above 15%. We also identified those with 
sampling errors below 10% as “high confidence” cases. We included some institutions with 
relatively low response rates (less than 25%) if they had a large number of respondents (more 
than 150), but we flagged such cases to be interpreted with caution. We reviewed exclusions 
carefully to avoid unintentionally biasing our sample (for example, because larger institutions 
tend to have lower response rates). From the remaining data, we created separate year-specific 
institutional scores for first- year and senior students. 

Because each institution had relatively few observations (a minimum of four and a 
maximum of nine for a given year in school), conventional time- series techniques were not 
suitable for the detection of trends. In addition, there is no theoretical reason to expect trends to 
take a particular functional form (e.g. linear). We wanted to make as few assumptions as possible 
about when a trend would begin and what shape it would take. Consider the range of possibilities 
for when improvement might begin: some campuses might implement change initiatives after 
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their first NSSE results, while others might wait until additional administrations confirm the need 
for improvement; some might take several years to design and implement a change effort; 
different change efforts might require different amounts of time before producing results; and 
some campuses may have had change efforts in place prior to their first NSSE administration, 
resulting in a left-censored trend. Next, consider the many possible shapes that a trend might 
take: some change efforts may show steady additive effects (linear growth); others might show 
strong initial effects that diminish over time (deceleration); still others may start small and build 
momentum over time (acceleration). Because these various scenarios of timing and patterning 
would affect the shape of the trend, we allowed for a range of possible functional forms. 

We ran trend analyses in Excel to generate goodness-of-fit R-square values for each 
measure by institution and class level (first-year or senior). We examined goodness-of-fit values 
for four different functional forms: linear, logarithmic, exponential, and power. We used the 
following criteria to determine a meaningful trend: 

• Statistically significant difference between first and last observation, with an effect 
size (Cohen’s d) of at least .3' 

• A pattern that fits at least one functional form with a goodness-of-fit R-square of at 
least .7 

After identifying institutions with positive and negative trends, we examined the patterns 
of change within institutions: Did trends focus on specific measures or combinations of 

* As recommended by Cohen (1988), we used a contextualized effect size criterion. NSSE developed a set of 
contextualized effect-size guidelines for the benchmarks (Gonyea & Sarraf, 2009), and we used the general 
suggestion of .3 for scales other than the percentage-based high-impact practices measure. 
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measures? Were trends limited to first-year or senior- year students? We also looked at the scales 
where trends were detected, to determine whether some measures evidenced more change than 
others. 

Phase Two: Understanding What Accounts for Change 

To conduct a more thorough examination of positive trends and what may account for the 
observed changes, we identified a subset of institutions representing a diverse group of 
institutional types from the larger dataset that showed strong positive trends (effect size of at 
least .4) across a variety of measures. We thus identified 142 institutions for further examination. 
The selected institutions had characteristics similar to the full dataset: 22% Doctorate-granting, 
52% Master’s, 23% Baccalaureate, and 3% other types of colleges and universities; 44% of the 
institutions were public and 6% were minority-serving institutions. With respect to 
undergraduate enrollment, 18% were very small (fewer than 2,000), 31% were small (2,000 - 
4,999), 22% were medium (5,000 - 9,999), and 30% were large (10,000-1-). 

Case study methods are well-suited for gaining an in-depth contextual understanding of a 
contemporary phenomenon, of organizational processes, and of the meaning of these experiences 
for those involved (Merriam, 1998; Yin, 2003). By investigating improvement efforts in a real- 
life context, we seek to promote a more dynamic and action-oriented perspective beyond simply 
cataloguing campus change initiatives (Yin, 1994). Ultimately, we seek to identify the specific 
characteristics, patterns, and practices that contributed to positive outcomes in order to produce a 
rich, thick representation of the complex realities of organizational learning and institutional 
change (Merriam, 1998; Sander, 1981). 
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To solicit institutional participation in phase two, we sent email messages to NSSE 
institutional contacts at selected institutions describing the project and how their institution was 
identified for further study, with a request to secure agreement to participate from the chief 
executive officer. Contacts at participating institutions were then asked to complete a 
questionnaire seeking their perspectives on the observed trends, information about change 
initiatives (if any), actors involved, and interpretations of what contributed to the results. We 
secured agreements to participate and completed questionnaires from 61 of the 142 institutions 
(43%). Of this group, 20% were Doctorate-granting, 56% Masters, 20% Baccalaureate, and 3% 
other types; 42% were public and 5% were minority serving institutions. The range of 
institutional characteristics thus remained fairly constant across the two phases of this study. 
Twenty institutions declined to participate, and nearly forty contacts did not respond to the 
invitation to participate. We followed up with about twenty of the contacts who either refused or 
did not respond, in order to understand their decision and any concerns they may have had. Most 
were apologetic, and explained that they were just too busy, with too many demands on the 
person(s) who would have to complete the questionnaire and conduct any follow-up. Contacts at 
two institutions said that they were unaware of the changes identified and were not interested in 
exploring what had occurred. 

The questionnaire included about a dozen questions, with a mix of closed- and open- 
ended questions on themes such as: awareness of the positive results, whether the institution had 
implemented any change efforts that the respondent believes had contributed to those results, 
motivation for and goals of change efforts, features of change efforts (for example, key actors, 
organizational units involved, etc.), whether assessment results informed the work, and what 
factors the respondent believed contributed to the positive results. We also invited respondents to 
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identify relevant documents for our review (e.g., assessment reports, strategic plans, 
accreditation self-studies, quality improvement plans, faculty senate minutes, and campus 
publications) and to suggest other possible informants. 

The research team then undertook textual analysis of questionnaire responses to identify 
key themes in the narratives of institutional change. Documents were also used to corroborate 
and augment evidence from the questionnaire (Yin, 2003). We conducted case analyses for each 
institution using questionnaire responses and additional documents and evidence provided by the 
institutional contact. We also consulted existing information that many of the institutions had 
previously reported (outside of the present research project) regarding the use of NSSE results on 
campus. 

The case analysis protocol included the following topics: What accounts for the observed 
positive change? What was the nature of the change effort (for example, was it goal- or mission- 
driven, motivated by assessment results, driven by external forces such as accreditation or 
legislative mandate, related to strategic planning, or unplanned and serendipitous?) To what 
extent was the change effort “home grown” versus based on established findings or examples 
from the field? What role did assessment data play in the initiative? If change was data-informed, 
what motivated the institution to pay attention to data? Who were key players or offices involved 
in change? We closely examined each case and characterized themes related to data use and 
institutional improvement, and we developed metaphors that characterized the change efforts. 
Members of the research team exchanged case analyses to achieve reliability and consistency 
across reviewers, and to consider alternate interpretations. We then conducted several extensive 
meetings to share findings from selected cases and discuss emerging themes. In a subsequent 
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phase of this project (beyond the scope of the present study), we will select a further subset of 
cases for site visits and more detailed case study analysis. 

Findings and Discussion 
Phase One Findings: Trends and Patterns 

Of 534 institutions, we found far more institutions with at least one positive trend (41 1) 
than at least one negative trend (72), providing strong evidence that our methodology is not 
simply detecting chance variation. This imbalance between positive and negative results is 
strongly suggestive of intentional efforts to improve the quality of undergraduate education. We 
found more instances of positive trends for first- year students (322) than for seniors (270) (table 

2). Negative trends were similarly distributed (44 for first- years and 38 for seniors), and tended 

2 

to be among institutions with higher initial scores. 

Patterns of change across measures and between first- year and senior scores suggest four 
propositions about institutional change. First, the first-year experience may be more amenable to 
change than the senior experience. The higher incidence of positive trends for first- year students 
suggests either that the first- year experience may be easier to change than the senior experience, 
that more institutions target the first year experience for improvement, or both. In either case, it 
is likely that the greater incidence of improvement in first- year measures reflects widespread 
concern for the first- year experience and student retention. This also corresponds to the 



^ The sum of institutions with positive (or negative) first-year and senior trends exceeds the total because some 
institutions had detectable trends for both first-years and seniors. 
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development of a robust body of knowledge about the first- year experience, including policies, 
strategies, programs, and services that facilitate a successful transition to college. 

The second proposition reflects an intentional and targeted focus of improvement efforts: 
institutions and faculty appear to be investing particular effort in promoting active & 
collaborative learning. For both first- year students and seniors, we found more positive trends 
for active & collaborative learning than for any other measure (table 2). In fact, the number of 
institutions that showed positive trends for first- year students on active & collaborative learning 
was at least twice that of all other measures, but for one. These patterns suggest broad efforts to 
adopt more engaging pedagogical practices that increase students’ involvement in learning. 

The second most common area of improvement was different for first- year students 
(student-faculty interaction) than seniors (supportive campus environment), but both of these 
suggest additional investments in promoting personal and supportive connections. Among 
seniors, this finding may indicate department-level efforts to improve advising and academic 
support, or to provide opportunities for students to develop meaningful relationships in the 
major. 



Indeed, many institutions showed positive trends for first- year students on both active & 
collaborative learning and student-faculty interaction. The combination of improvement across 
these two benchmarks suggests the possible influence of the First Year Experience movement 
(Upcraft & Gardner 1989), which has focused attention on the importance of providing new 
students an orientation to college via a challenging and academic first- year experience that 
promotes student- faculty interaction and quality interaction with peers. Concerted efforts have 
been made to enhance the first- year experience through a comprehensive body of research and 
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evidence-based practice on the first year of college, and since the 1980s through national 
convenings like the First Year Experience conference. 

We found many instances of positive trends on the same measure for both first- year 
students and seniors. This pattern reflects a third proposition about change: many institutions 
appear to have endorsed a particular broad-based change in a way that spans class levels. A 
broad improvement agenda might be reflected in, for instance, positive trends in supportive 
campus environment scores for both first- year students and seniors. The increase might be 
associated with institution- wide efforts to improve academic support services or implement 
expanded, integrated, and better coordinated academic support services, or the creation of a “One 
Stop Shop” for student support services such as financial aid, admission, bursar, registrar, and 
career services. 

We found positive trends across the spectrum of institutional differentiation (control, 
size, Carnegie type). This finding leads to a fourth proposition that challenges the conventional 
wisdom about the relationship between institutional characteristics and change: capacity for 
sustained, positive change is not limited to small colleges, private institutions, or residential 
institutions. For example, the active & collaborative learning benchmark accounted for the most 
instances of change (table 2), and also the most cases with larger effect sizes (.5 or greater), 
across the range of institutional types. We found many instances of positive trends in active & 
collaborative learning in the first year, including at large public institutions. Contrary to the 
common belief that urban or commuter institutions do not typically support high levels of 
student-faculty interaction, we found that 41% of the institutions with positive trends on this 
benchmark for first- year students are urban institutions. Urban and commuter institutions 
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showing improved scores may have adopted pedagogical practices and policies that promote 
student-faculty interaction. The fact that we found patterns of systematic positive change at both 
public and private institutions, in every size category and Carnegie type, suggests that the 
potential for improving performance exits across the full range of colleges and universities. 

Although these results suggest the promise of reform, it is also important to consider the 
measures where we found the fewest positive trends. For first- year students, positive trends on 
diversity experiences and support for academic success were least common. Among seniors, the 
lowest frequency was for high-impact practices and integrative learning experiences, followed by 
student-faculty interaction and diversity experiences. Several phenomena may be at work here, 
such as amenability, attention, and ceiling effects. Some may be genuinely difficult to improve 
relative to the other measures in our study, and this difficulty manifests itself in the low 
frequency of positive trends. Another possible factor is attention — the processes tapped by the 
measure may be relatively less important to faculty and administration, and consequently there 
are fewer efforts to improve it. Finally, ceiling effects are possible: if baseline performance is 
already relatively high, the opportunity for realizing improvement — especially sufficient to be 
detectable as a trend — may be quite limited. 

For the measures noted above for having a low incidence of positive trends, there is little 
evidence that this is due to ceiling effects. If baseline performance is high, we would expect 
more opportunities for decline. But table 2 shows that the measures noted above are generally 
not the ones with a high incidence of negative trends. The one possible exception is support for 
academic success, which is tied for the highest incidence of negative trends among first- year 
students. But at only nine such cases compared to 55 cases of positive trends, the evidence is 
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weak. Further examination of aggregate responses from a recent NSSE administration on the 
component items of this measure show ample room for improvement: only 35% of first- year 
students who reported “very much” institutional emphasis on providing support for academic 
success, 33% who rated academic advising as “excellent,” and 36% and 48%, respectively, who 
rated the quality of relationships with administrators and faculty as a 6 or 7 on the 7-point scale. 

That leaves amenability and attention as possible explanations. Because two of three 
components of our diversity measure are based on the frequency of serious conversations with 
different others, it may be that institutions have limited ability to directly influence this measure. 
To achieve gains in high-impact practice participation, at least one if not both of the following 
must occur: they must be made available to more students, and more students must avail 
themselves of the opportunities. Institutions may face challenges on both fronts. But given that 
involvement in high-impact practices such as service-learning, study abroad, undergraduate 
research, and culminating senior experiences corresponds to desirable outcomes — especially for 
traditionally underserved populations (Kuh, 2008) — this reflects an important dimension for 
institutional growth and there may be great value in learning how some institutions managed to 
achieve steady gains. 

The small number of institutions that saw improvement in integrative learning may 
reflect the challenge of creating a coherent educational program in which students frequently 
participate in academic activities that integrate ideas from various sources, include diverse 
perspectives, and discuss ideas with others outside of class. There is room to bring all these 
programs up to scale at all institutions and thereby greatly increase their transforming effects on 
students’ lives and learning. 
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Phase Two Findings: Accounting for Change 

These aggregate findings are tantalizing. To advance higher education’s improvement 
agenda, however, we need to move beyond description to understand the conditions of change at 
the institutional level. By examining each positive trend institution as a case, we can get a more 
in-depth and nuanced account of improvement over time that can advance our understanding of 
institutional change and the potential for gauging the impact of reform efforts. 

Results from the questionnaire revealed important information about the extent to which 
the trends that we observed in the NSSE data were the result of intentional change efforts, and 
about the genesis of those efforts. Importantly, all but four of the 61 institutional contacts 
reported that they had implemented change efforts that they believe account for the positive 
results. Three were unsure, but provided a comprehensive accounting of campus change efforts, 
and one indicated that no intentional change had been undertaken on the campus. We asked 
respondents to identify all the motivators for their change efforts from a fixed response set based 
on a recent National Institute for Learning Outcomes Assessment (NILOA) study of campus 
assessment and improvement initiatives (Kuh & Ikenberry, 2009). Nearly all respondents 
identified an “institutional commitment to improving undergraduate education” as one of the 
motivators behind their change efforts. While in part this may be attributable to retrospective 
sense making, it also evokes the observation by Kuh et al. (2005) that a “positive restlessness” 
around student learning and undergraduate education existed at institutions with better-than- 
expected levels of student engagement. 

The next most popular response was “data that revealed concerns about undergraduate 
education.” This is highly consistent with the perspectives on the genesis of organizational 
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learning articulated by Argyris and Schon (1978), Levitt and March (1988), and Schein (1996) 
involving unfulfilled aspirations or dissatisfaction with performance. 

The third most frequently cited motivation was “faculty or staff interest in improving 
undergraduate education.” Although several campuses indicated other response options as 
influential, including “accreditation” and “internal program review,” it was noteworthy that 
“national calls for accountability,” and “mandates from governing, state or legislative boards” 
were identified by very few respondents as motivating change initiatives. 

Elaborations on what motivated campus change efforts revealed a variety of different 
issues. However, the common catalyst for change was clearly an institutional commitment to 
improvement plus data corroborating a specific concern. At a private Eastern doctoral university, 
the president charged the core curriculum committee to be “bold, not old” in plans to address 
concerns about a stale, incoherent curriculum and declines in retention. Ten years of data on a 
variety of student learning measures did not align with strategic goals, so action was needed. An 
Eastern master’s institution’s commitment to improving the first- year experience, coupled with a 
tradition of assessing incoming students’ academic readiness and concerns about gateway course 
success rates, led to an enriched orientation program and pedagogical changes in first- year 
courses. 

These results about catalysts for change show the strong influence of intrinsic motivation, 
internal constituencies, and evidence about quality in the undergraduate experience. While the 
NIEOA study found that accreditation was the main driver for assessment activities in colleges 
and universities (Kuh & Ikenberry, 2009), our results showed accreditation was a less important 
force behind change initiatives. While initially surprising, this finding highlights the difference 
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between assessment that is undertaken for accountability and compliance purposes and 
assessment that informs institutional diagnosis and improvement. Finally, the limited influence 
of external forces such as national calls for accountability and state mandates for improvement 
suggest these forces have limited capacity to trigger genuine change and improvement. 

The next theme relates to what facilitated change efforts. An unsurprising finding was 
that leadership from the top — by presidents and provosts — was important to bringing about 
change and sustaining the improvement agenda over time. The importance of senior leadership 
cannot be overstated. According to our informants, the most important factor related to success 
of a change initiative was visible, sustained commitment from campus leadership — trustees, 
presidents, deans, and faculty. An institutional research director at a Southern university credited 
“clear endorsement by leaders in administration that maintained momentum. . . . Adequate 
resources for faculty and staff reinforced the administrative message that the improvement 
initiatives are valued.” A private, urban institution illustrated the importance of leadership to 
evidence-driven improvement. The institution had launched an extensive examination of the 
quality of service delivery and quality of campus relationships. Administrative leaders were 
already attuned to the need to reduce bureaucracy and develop more “one-stop” services, and 
analyses of longitudinal NSSE results on the quality of campus relationships and satisfaction 
measures reinforced this need. Institutional leaders promoted the assessment project and then 
invested attention and resources in the restructuring of student services, particularly the registrar, 
bursar, and financial aid operations. 

Another important finding related to the facilitation of change is participation in national 
programs and initiatives. Examples include the Eoundations of Excellence (EOE) self-study and 
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improvement process of the John N. Gardner Institute for Excellence in Undergraduate 
Education; the Teagle Eoundation’s project to improve student learning; the Wabash National 
Study of Eiberal Arts Education; and the Association of American Colleges and Universities’ 
(AAC&U) Bringing Theory to Practice (BTP) and Eiberal Education and America’s Promise 
(EEAP) projects. A mid- Atlantic master’s institution participating in both the POE and BTP 
initiatives saw its results over seven NSSE administrations show improvement on several first- 
year measures, including active and collaborative learning and student-faculty interaction. This 
institution credited its intensive work with the POE self-study to identify where students were 
encountering obstacles in the first year, and the BTP project for providing a rich framework for 
considering evidence-based practice to improve student success. To improve the first- year 
experience and retention rates, the institution created “road maps” to aid students’ academic 
planning and completion of requirements, increased its emphasis on learning communities and 
first- year seminars, implemented a peer-to-peer supplemental instruction program for first- year 
students, and provided an opportunity for students to “catch up” though hybrid and online 
courses offered to first-year students during winter and summer breaks. Our results indicate that 
participation in initiatives like POE and BTP provided institutions with structure and support to 
explore assessment results and make changes in practice informed by both local assessment 
results and best practices in the field. 

After noting several institutions that reported on and credited their involvement in 
national initiatives as initiating and often sustaining their improvement efforts, we wondered 
whether other institutions identified for the present study were also involved in such initiatives. 
Many initiatives publicize lists of institutional participants, so we were able to tabulate the 
proportion of institutions that had participated selected projects. Table 3 reports the percentage of 
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institutions invited to participate in the present study that were involved in three national 
initiatives, and also the percentage of all initiative participants invited for this study. The strong 
representation in this study of institutions involved in these projects suggests a beneficial 
relationship, especially for the Foundations of Excellence project, with 44% of its participants 
having been identified in the present study as having positive trends on one or more of the 
measures examined. 

Although accreditation in itself was not widely identified as a driver for change efforts, 
many respondents indicated that their change efforts were related to specific quality- 
improvement aspects of regional accreditation. A private institution indicated that the Southern 
Association of Colleges and Schools’ Quality Enhancement Plan (QEP) was instrumental in 
advancing a project to increase experiential learning. The project led to efforts to promote 
foreign language courses, study abroad, and increased opportunities for global learning and 
diversity experiences across the educational program. However, it was the structure and 
requirements of the QEP process that helped the institution advance its objectives. The long-term 
process of developing institutional investment in the QEP and its implementation, and then the 
expectation for the regular collection of evidence to evaluate program effectiveness and assess 
participation and continuous improvement, provided the institution the support to implement, 
refine, and sustain the project. 

Meaningful strategic planning processes were also an important factor in change efforts. 
A large, public commuter institution in the Midwest credited strategic plan goals to “enhance 
student access to and successful participation in higher education through quality and innovative 
instruction” and to provide “student life programs that increase graduation rates and provide 
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career placement opportunities for a diverse student body” with orienting their reform efforts. 

The plan promoted several promising programs on campus, including academic and co-curricular 
programs supporting African-American student retention. The plan also helped advance their 
efforts to develop more service-learning experiences. Most importantly, the strategic plan 
provided a framework for launching an array of institutional improvement initiatives to improve 
student success. 

The need to address a real campus problem, such as declining retention and graduation 
rates or a financial crisis, or to address other concerns revealed by data facilitated change at 
many institutions. For example, data regarding the number of hours of off-campus work and 
corresponding low levels of interaction with peers, coupled with declining persistence rates, 
signaled a concern to a small private college in the Midwest. The institution compiled additional 
evidence and sought external funds to underwrite several changes in the first year experience. 

The resulting small grant-funded pilot project garnered positive attention. The grant, the local 
attention it received, and evidence of the program’s effectiveness helped institutionalize the 
reform. 



These accounts offer initial insight into activities on the ground that led to the changes 
observed in longitudinal NSSE results. We are continuing to analyze questionnaire results in 
preparation for in-depth site visits to selected campuses in the next phase of this study. 

Limitations 

This study is not without its limitations, of course. By selecting institutions with positive 
trends, we risk making stronger post-hoc attributions than might be justified — our study design 
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does not permit identification of institutions that may have implemented similar change efforts 
that did not yield positive results. When asked to account for positive results, our informants may 
be subject to retrospective sense-making, constructing plausible accounts for the positive results. 
We nevertheless find that our data conform to conceptual accounts of organizational learning, 
and by studying a large number of cases we are able to identify common patterns in informants’ 
accounts that lend support to our interpretations. 

Summary and Conclusion 

NSSE was created to inform institutional improvement efforts. After a decade in the 
field, we can begin asking important questions about whether and how colleges and universities 
can improve student engagement. A careful analysis of time series data for 534 institutions that 
administered NSSE from four to nine times between 2001 and 2009 revealed an appreciable 
number of institutions with detectable trends. Positive trends far outnumbered negative ones, by 
a margin of about 7:1. Our examination of change statistics across a range of measures of 
effective educational practice, deep approaches to learning, and high-impact practices 
demonstrate that it is possible to “move the needle” with regard to student engagement. An 
important finding of this work is that improvement is possible across the spectrum of 
institutional diversity. 

Eollowing the quantitative analysis of trends and patterns within those trends, we 
undertook qualitative inquiry at a diverse group of institutions with strong positive trends in 
order to illuminate the circumstances behind the observed trends. Our analysis suggests that most 
improvement efforts come not as a result of external pressure and accountability demands, but 
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from an intrinsic motivation to improve, often motivated by data that revealed a gap between 
aspirations and actual performance. 

This paper offers plentiful existence proofs that intentional change is possible in higher 
education. It advances our understanding of change in higher education institutions and offers 
practical insights for reform-minded institutions. It contributes to the national conversation about 
how institutions can improve performance to address widespread concerns about quality and 
success in undergraduate education. Higher education institutions operate in a challenging 
climate of high expectations, diminished resources, and serious questions — even suspicion — 
about commitment to educational value. There is perhaps no better time to tell an evidence-based 
story about institutional improvement and for colleges and universities to effectively demonstrate 
their commitment to improving student engagement and learning. 
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Table 1 — Number and percentage distribution of institutions according to number of NSSE 
administrations between 2001 and 2009 





Four 


Five 


Six 


Seven 


Eight 


Nine 


Total 


Number 


175 


124 


97 


53 


51 


34 


534 


Distribution 


33% 


23% 


18% 


10% 


10% 


6% 


100% 



Table 2 — Number of institutions with trends detected, by criterion measure, trend 
direction, and class level 




Positive 

Eirst 

year 


trends 

Senior 


Negative trends 
Eirst 

year Senior 


Academic challenge 


53 


41 


7 


6 


Active & collaborative learning 


147 


116 


2 


5 


Active learning 


144 


75 


3 


4 


Collaborative learning 


53 


47 


3 


2 


Student-faculty interaction 


73 


33 


1 


2 


Course-related interactions with faculty 


78 


44 


2 


3 


Out-of-class interactions with faculty 


46 


26 


3 


4 


Supportive campus environment 


57 


96 


4 


10 


Support for success and enrichment 


62 


56 


4 


1 


Support for academic success 


32 


46 


9 


14 


Courses emphasize higher-order thinking 


55 


36 


0 


4 


Courses emphasize integrative learning 


50 


29 


2 


4 


Experiences with diversity 


20 


34 


8 


5 


High-impact practices 


55 


26 


9 


3 


Across measures, at least one trend detected 


322 


270 


44 


38 



Table 3 — Participation in Three National Initiatives by Institutions in the Present Study* 





Eoundations 
of Excellence 


Bringing Theory 
to Practice 


EEAP 


Number of study institutions in initiative 


48 


16 


30 


Share of all study institutions (N=140) 


34.3% 


11.4% 


21.4% 


Total number of initiative institutions 


no 


123 


310 


Share of initiative institutions 


43.6% 


13.0% 


9.7% 



*As of February, 201 1 
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Appendix 



Description of Measures 

Below are brief definitions and item specifications for the 14 measures examined in the study. For 
response options and additional information about these items on the NSSE survey, please refer to the 
NSSE codebook: www.nsse.iub.edu/2009 Institutional Report/pdf/NSSE%202009%20Codebook.pdf 

Level of Academic Challenge 

The extent to which an institution promotes high levels of student achievement by emphasizing 
the importance of academic effort and setting high expectations for student performance. 



Variable Description 



readasgn 

writemor 

writemid 

writesmi 

analyze 

synthesz 

evaluate 

applying 

workhard 

acadprOl 

envschol 



Number of assigned textbooks, books, or book-length packs of course 
readings 

Number of written papers or reports of 20 pages or more 

Number or written papers or reports between 5 and 19 pages 

Number of written papers or reports of fewer than 5 pages 

Analyzing the basic elements of an idea, experience, or theory, such as 
examining a particular case or situation in depth and considering its 
components 

Synthesizing and organizing ideas, information, or experiences into new, 
more complex interpretations and relationships 

Making judgments about the value of information, arguments, or methods, 
such as examining how other gathered and interpreted data and assessing 
the soundness of their situations 

Applying theories or concepts to practical problems or in new situations 

Worked harder than you thought you could to meet an instructor's 
standards or expectations 

Preparing for class (studying, reading, writing, doing homework or lab work, 
analyzing data, rehearsing, and other academic activities) 

Spending significant amounts of times studying and on academic work 
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Active & Collaborative Learning 

How often students participate in class and collaborate with other students in solving problems or 
mastering difficult material. 



Variable 


Description 


clquest 


Asked questions in class or contributed to class discussions 


cipresen 


Made a class presentation 


classgrp 


Worked with other students on projects during class 


occgrp 


Worked with classmates outside of class to prepare class assignment 


tutor 


Tutored or taught other students (paid or voluntary) 


commproj 


Participated in a community-based project (e.g., service learning) as a part 
of a regular course 


oocideas 


Discussed ideas from your readings or classes with others outside of class 
(students, family members, co-workers, etc.) 



Active Learning 

Students learn more when they are intensely involved in their education and are asked to think 
about and apply what they are learning in different settings. 



Variable 


Description 


clquest 


Asked questions in class or contributed to class discussions 


cipresen 


Made a class presentation 


commproj 


Participated in a community-based project (e.g., service learning) as part of 
a regular course 
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Collaborative Learning 

Collaborating with others in solving problems or mastering difficult material prepares students to 
deal with the messy, unscripted problems they will encounter daily during and after college. 



Variable 


Description 


classgrp 


Worked with other students on projects during class 


occgrp 


Worked with classmates outside of class to prepare class assignments 


tutor 


Tutored or taught other students (paid or voluntary) 


oocideas 


Discussed ideas from your readings or classes with others outside of class 
(students, family members, co-workers, etc.) 



Student-Faculty Interaction 

How often students interact with faculty members inside and outside the classroom. 



Variable 


Description 


facgrade 


Discussed grades or assignments with an instructor 


facideas 


Discussed ideas from your readings or classes with faculty members outside 
of class 


facpians 


Talked about career plans with a faculty member or advisor 


facfeed 


Received prompt written or oral feedback from faculty on your academic 
performance 


facother 


Worked with faculty members on activities other than coursework 
(committees, orientation, student life activities, etc.) 


resrch04 


Worked on a research project with a faculty member outside of course or 
program requirements 
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Course-Related Interactions with Faculty 

The extent to which students interact with faculty on course-related matters. 



Variable 


Description 


facgrade 


Discussed grades or assignments with an instructor 


facideas 


Discussed ideas from your readings or classes with faculty members outside 
of class 


facfeed 


Received prompt written or oral feedback from faculty on your academic 
performance 



Out-of-Class Interactions with Faculty 

The extent to which students interact with faculty in out-of-class activities. 



Variable 


Description 


facpians 


Talked about career plans with a faculty member or advisor 


facother 


Worked with faculty members on activities other than coursework 
(committees, orientation, student life activities, etc.) 


resrch04 


Worked on a research project with a faculty member outside of course or 
program requirements 
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Supportive Campus Environment 

The extent to which students perceive the institution is committed to their success, and provides 
institutional support for academic success, and cultivates high quality student relationships with 
faculty and administrators, and peers. 



Variable 


Description 


envsocal 


Providing the support you need to thrive socially 


envsuprt 


Providing the support you need to help you succeed academically 


envnacad 


Helping you cope with your non-academic responsibilities (work, family, 
etc.) 


envstu 


Quality of relationships with other students 


envfac 


Quality of relationships with faculty members 


envadm 


Quality of relationships with administrative personnel and offices 



Support for Success and Enrichment 

The extent to which students perceive the institution is committed to their success and cultivates 
positive working and social relations among different groups on campus and opportunities to 
enhance academic enrichment. 



Variable 


Description 


envschol 


Spending significant amounts of time studying and on academic work 


envsuprt 


Providing the support you need to help you succeed academically 


envcompt 


Using computers in academic work 


envdivrs 


Encouraging contact among students from different economic, social, and 
racial or ethnic backgrounds 


envnacad 


Helping you cope with your non-academic responsibilities (work, family, 
etc.) 


envsocal 


Providing the support you need to thrive socially 


envevent 


Attending campus events and activities (special speakers, cultural 
performances, athletic events, etc.) 
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Support for Academic Success 

The extent to which students perceive the institution is committed to their success, cultivates 
quality relationships among students and faculty and administrators, and promotes quality 
advising. 



Variable 


Description 


envsuprt 


Providing the support you need to help you succeed academically 


envfac 


Quality of relationships with faculty members 


envadm 


Quality of relationships with administrative personnel and offices 


advise 


Qverall evaluation of the quality of academic advising you have received at 
your institution 



Deep Approaches to Learning: Higher Order Thinking 

The extent to which students believe that their courses emphasize advanced thinking skills such as 
analyzing the basic elements of an idea, experience, or theory; and synthesizing ideas, information, 
or experiences into new, more complex interpretations. 



Variable 


Description 


analyze 


Analyzing the basic elements of an idea, experience, or theory, such as 
examining a particular case or situation in depth and considering its 
components 


synthesz 


Synthesizing and organizing ideas, information, or experiences into new, 
more complex interpretations and relationships 


evaluate 


Making judgments about the value of information, arguments, or methods, 
such as examining how others gathered and interpreted data and assessing 
the soundness of their conclusions 


applying 


Applying theories or concepts to practical problems or in new situations 
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Deep Approaches to Learning: Integrative Learning 

How often students participate in academic activities that integrate ideas from various 
sources and include diverse perspectives, and discuss ideas with others outside of class. 



Variable 


Description 


integral 


Worked on a paper or project that required integrating ideas or 
information from various sources 


divciass 


Included diverse perspectives (different races, religions, genders, 
political beliefs, etc.) in class discussions or writing assignments 


intideas 


Put together ideas or concepts from different courses when 
completing assignments or during class discussions 


facideas 


Discussed ideas from your readings or classes with faculty members 
outside of class 


oocideas 


Discussed ideas from your readings or classes with others outside of 
class (students, family members, co-workers, etc.) 



Experiences with Diversity 

How often students interact with other students who are different from them in terms of 
race, ethnicity, religious beliefs, political opinions, and personal values; and the extent to 
which the institution encourages this interaction. 



Variable 


Description 


divrstud 


Had serious conversations with students of a different race or 
ethnicity than your own 


diffstu2 


Had serious conversations with students who are very different from 
you in terms of their religious beliefs, political opinions, or personal 
values 


envdivrs 


Encouraging contact among students from different economic, social, 
and racial or ethnic backgrounds 
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High-Impact Practices 

The proportion of students who report participation in specific educational experiences 
that have been shown to relate positively to student outcomes. The relevant population 
(first-years [FY] or seniors [Sr]) is identified in parentheses. 



Variable 


Description 


Irncom04 


Done: Participated in a learning community or some other formal 
program where groups of students take two or more classes together 
(FY) 


com m pro j 


Participated in a community-based project (e.g., service learning) as 
part of a regular course (FY & Sr) 


stdabr04 


Done: Study abroad (Sr) 


snrx04 


Done: Culminating senior experience (capstone course, senior project 
or thesis, comprehensive exam, etc.) (Sr) 


inter04 


Done: Practicum, internship, field experience, co-op experience, or 
clinical assignment (Sr) 


resch04 


Done: Worked on a research project with a faculty member outside 
of course or program requirements (Sr) 
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